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DATA ENTRY METHOD AND SYSTEM 


Related Application Data 
This application claims nFfority to copending application 09/543,125, filed April 5, 2000, 
which claims priority to provisional patent application 60/163,332, filed November 3, 1999. 


Field of the Invention 
The present invention relates to optical sensing of digital data, and use of such 
technology in a variety of contexts (e.g., cell phones) and for a variety of purposes (e.g., 
1 0 merchandise ordering) . 


Background and Summary of the Invention 
In copending applications 09/343,104 (filed June 29, 1999), and 09/292,569 (filed April 
1999), the preset assignee details a variety of technologies that employ optical data sensing 
15 to invoke various communication or control functions. The present specification builds on those 
earlier applications, detailing additional novel technologies and systems. 

For example, according to one aspect of the present invention, a wireless telephony 
device is equipped with an optical image sensor, and a lens for imaging an object onto the 
sensor. The optical sensor permits a variety of useful functionality, including decoding of bar 
20 codes, watermarks, etc., from objects imaged by the sensor. 

One application of such a device is to interact with a remote computer system using data 
decoded from a bar code or watermark. For example, the device can be used to scan image data 
from an item depicted in a catalog, and then to transmit an order for the item from the catalog's 
on-line web store - using the associated digital data decoded from the image to identify the item 
25 desired. 

The foregoing and additional features and advantages of the present technology will be 
more readily apparent from the following detailed description. 
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Detailed Description 

The cited '104 application details a variety of optical input devices, including some in 
which the optical sensor element and "Bedoop" (now known as Digimarc MediaBridge) 
decoding circuitry are integrated on a common substrate. A variety of other novel input devices 
5 can also be used. 

One is a mouse-like peripheral that includes an optical sensing system. The optical 
sensing system can comprise a ID array of plural optical sensors (e.g., CCD, CMOS, etc.), or a 
2D array. Such devices are already known in other contexts, e.g., the Microsoft IntelliMouse 
with IntelliEye technology. That device includes a multi-element CMOS optical sensor 
10 integrated on an IC with various detector and processing circuitry, operating in conjunction with 
a short focal length imaging lens and an LED illumination source. The circuitry tracks 
movement of patterns across the sensor's field of view, by which the mouse's movement can be 
deduced. The Microsoft product collects 1500 data sets per second - a rate much higher than is 
needed in most embodiments of the assignee's Bedoop technology. 
15 Such a mouse-like peripheral can omit the buttons and position-sensing features 

commonly provided on traditional mice, yielding a simple desk-facing palm camera that 
. generates frames of data corresponding to a small area under the sensor portion of the mouse. 
More typically, however, the peripheral includes the buttons, roller wheels, and/or X-/Y - position 
sensing arrangements of traditional mice so that button and positional forms of data input can be 
20 exploited in interacting with the Bedoop application. 

The optical data collected by the sensor can be processed within the peripheral's 
processor to extract the steganographically encoded binary Bedoop data therefrom. Or this 
processing burden can be undertaken by the associated computer system, with the peripheral 
simply processing and formatting the raw sensor data into sequential frames of image data to be 
25 output to that system. 

Any form of hand-held scanner - whether of the type just described or others known in 
the art - offers a convenient way to interact with catalog advertising. Imagine a traditional paper 
catalog, e.g., from L.L. Bean, Inc., or Lands End. Each image in the catalog is Bedoop-encoded 
with a code that identifies the depicted product. A user browsing through the catalog, on seeing 
30 a product of interest, places the scanner over the picture (and optionally may be required to push 
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a button or otherwise signal to initiate further processing). The scanner detects the Bedoop data 
and relays it to an associated computer (optionally with data identifying the consumer). The 
computer polls a remote server computer maintained by the merchant, which responds with data 
corresponding to the item depicted in the scanned image. This returned data can include data 
indicating the sizes available, data indicating the colors available, data indicating the variant 
styles available, flag bits indicating whether each item is in stock, etc. This returned data can be 
presented to the consumer - typically on a display device but alternatively in audible form. 

Preferably, the customer's body measurements (waist size, inseam length, neck size, etc.) 
are stored in a user profile, either on the local computer or at the merchant's server computer. 
This allows the system to customize the data presented to the user - e.g., showing the color 
options and availability only for the depicted shirt in a 16 inch neck and a 34 inch sleeve length. 

If necessary, the user can select among the color or style options, using the handheld 
input device (either buttons, gestures, etc.), or any other input device. Or the item may be one for 
which no further specifications are needed. In either event, once the desired product has been 
sufficiently specified, the user can signal the system to place the order. Payment and shipping 
details can be arranged through any of the great variety of techniques known in the art, e.g., by 
charging to a credit card number and shipping to an address on-file with the merchant (as 
detailed, e.g., in Patent 5,960,41 1). 

While scanning peripherals of the type described above are typically wired to an 
associated host system, wireless links (e.g., radio, infrared, ultrasonic, etc.) can of course be 
used, freeing the user from the constraint imposed by the cable. 

One of the embodiments detailed in the earlier applications is a pay telephone with a 
Bedoop sensor. The technology aan likewise be integrated into cellular phones, of the type 
available from Motorola, Nokia/Qualcomm, and others, and reviewed in applicant's allowed 
application 09/172,324. Such ^phone can be equipped with a ID or 2D image sensor, the output 
of which is applied to Bedoop decoding circuitry within the phone. This decoding circuitry can 
be the phone's main CPU, or can be a processing circuit dedicated to Bedoop functionality. (As 
noted elsewhere, the decoding can be effected by dedicated hardware, by decoding software 
executing on a general purpose CPU, etc.) 
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Cell phones are already equipped with numerous features that make them well suited for 
Bedoop operation. One is that cell phones typically include an LCD or similar screen for display 
of textual or graphic information, and additionally include buttons or other controls for selecting 
among menu options presented on the screen (e.g., by moving a cursor. Moreover, cell phones 
5 naturally include both audio input and output devices (i.e., microphone and speaker). Still 

further, the protocol by which cell phones transmit data includes data identifying the phone, so 
that such data need not be separately encoded. And finally, cell phones obviously provide ready 
links to remote computer systems. Collectively, these capabilities rival those of the most fully- 
equipped desktop computer system. Thus, essentially all of the applications detailed in the prior 
10 applications can be implemented using cell phone Bedoop systems. 

As with the other Bedoop systems, when Bedoop data is sensed, the phone can respond to 
the data locally, or it can forward same over the cellular network to a remote system (or 
computer network) for handling. 

One application that may be invoked locally (i.e., within the phone) is the dialing of a 
15 corresponding telephone number (as detailed, e.g., in the above-cited applications). In some 
embodiments, the phone number is literally encoded as part of the Bedoop data payload. In 
others, the phone number is stored in a memory, within the phone, and indexed in accordance 
with an identifier decoded from the Bedoop data. 

The variety of operations that can be handled remotely is virtually limitless. Some entail 
20 interaction with the user. For example, the remote system may initially respond by presenting to 
the user a menu of options on the display screen (e.g., Purchase, Add to Shopping List, Request 
Sample, Add to Notepad, etc.) The user then responds by providing further input (e.g., by 
manipulating one or more buttons on the phone, by providing spoken instructions to a voice 
recognition sub-system within the phone, etc.). This further data is then dispatched from the 
25 phone, and the requested action undertaken. Other operations don't require further interaction 
with the user, but immediately invoke a corresponding action. 

In this and other contexts, it will be recognized that the gestural input concepts detailed in 
the prior applications involve relative movement between the sensor and the encoded object. In 
most of the earlier examples, the sensor is stationary, so gesticulation is effected by moving the 
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object. Naturally, if the sensor is moveable (e.g., as a cell phone), the gestural movement can be 
effected by moving the sensor instead. 

In the earlier-filed applications, the user can solicit different responses by moving the 
object relative to the sensor. In other embodiments, different responses can be solicited without 
5 movement, e.g., by presenting the object at different orientations. 

Consider a magazine advertisement. When presented to the sensor with the top of the 
page up, a first response can be invoked. If the page is presented at a rotation of 90 degrees, a 
second response can be invoked. Similarly with 180 degrees rotation (i.e., upside down), and 
270 degrees rotation. The Bedoop detector can detect these different rotational states by 
10 reference to attributes of the watermark signal discerned from the magazine page (e.g., by 

reference to the rotation state discerned from the subliminal grid signal detailed in applicant's 
prior patents). 

While the just-described arrangement included the Bedoop decoding function within the 
phone, in other embodiments the image data can be transmitted from the phone and decoded at a 

15 remote location. 

In the earlier-filed applications, reference was made to GPS receivers as permitting the 
location of a person to be tracked, and contact information updated accordingly. GPS receivers 
find many other applications in Bedoop contexts. For example, the response of a Bedoop system 
can be tailored, or vary, in accordance with the location of the person invoking the operation. To 

20 illustrate, if a user presents a newspaper insert or coupon for a Dominos pepperoni pizza meal to 
the Bedoop sensor on a GPS-equipped cellular phone, the GPS data can be encoded in the 
Bedoop data dispatched to the Domino's Bedoop server. That server can identify, by reference 
to the GPS data, the location of the nearest Domino's franchise, and can forward the order to that 
franchisee. The franchisee, in turn, can telephone the user (by reference to telephone number 

25 data from the cell phone) to confirm the order, inquire if additional items are desired, inform the 
user of the final price, and to obtain the delivery address. (The latter step can be omitted; the 
franchisee can employ the GPS data to obtain the corresponding street address, e.g., from map 
data licensed through NavTech of Chicago.) 


30 conveyed from the phone to the cellular network can take various forms; the design of such 


The protocols by which the Bedoop data, GPS data, and cell phone identification data are 
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systems are familiar to those in the relevant arts. In one embodiment, the protocols by which 
some cell phones are now provided with email or internet access are further adapted to convey 
Bedoop and GPS data. The protocols for conveying telephone identification data are already 
well established. Once received by the cellular network, the Bedoop data can be formatted into 
5 packets, either incorporating the GPS data and telephone data into the same packet structure, or 
by formatting such data into separate packets accompanying the Bedoop packets. 

The provision of image sensors in cell phones enables other functionality. One is the 
capture of still or video imagery. Such image data can be compressed (typically by lossy 
processes such as MPEG, JPEG, or the like, implemented with dedicated hardware CODECs) 

10 and transmitted with the audio data. The screens on such phones can likewise be used for 
display of incoming image or video data. 

Another function enabled by image sensors in cell phones is user-verification, e.g., by 
retinal scanning or other optically-sensed biometrics, before the phone will permit a call to be 
placed. A great number of such biometric verification techniques are known. 

1 5 Cell phone Bedoop sensors may not always be in communication with the cell phone 

network. The phone may be out of range of a cell site, or. may be in operational mode in which 
an RF link is not then established. In such events, any Bedoop data sensed by the phone that is 
to be handled remotely is desirably stored locally within the phone, and queued for transmission 
to the cellular network the next time communication is established (a so-called "store and 

20 forward" form of operation). 


Wholly unrelated to the foregoing is the notion of a Bedoop-encoded tattoo. Temporary 
tattoos are well known and typically include an ink or printed film that is transferred to a 
25 wearer's skin through some application process (e.g., wetting or otherwise). The artwork on the 
tattoo can be arranged to steganographically encode Bedoop data. 

E-Paper 

While the prior applications noted that a great variety of printing technologies can be 
30 employed in Bedoop applications, it should be noted that e-paper can be employed as well. E- 


Tatoos 




WYC:lmp 60244 


-7- 


EXPRESS MAIL EL525675926US 


paper, developed by Nicholas K. Sheridon of Xerox, and mass produced by 3M, is a thin layer of 
transparent plastic in which millions of small beads, somewhat like toner particles, are dispersed. 
The beads, each contained in a fluid -filled cavity, are free to rotate within those cavities. The 
beads are "bichromal," with hemispheres of contrasting color (e.g. black and white). Under the 
5 influence of a voltage applied to the surface of the sheet, the beads rotate to present one colored 
side or the other to the viewer. A pattern of voltages can be applied to the surface in a bit- wise 
fashion to create images such as text and pictures. The image persists until new voltage patterns 
are applied to create new images. The reader is presumed familiar with the US patents issued to 
Sheridon on this technology. 


It will further be recognized that epaper can be used to convey digital data according to 
essentially any known watermarking method, including those detailed in the patents and 
applications incorporated by reference, and is also suitable for conveying digital information in 
data glyph form. 


Revenue Sharing 

When a consumer presents a Bedoop-encoded object to a sensor, and as a result of the 
link(s) thereby established, purchases a product or service, the revenue from that transaction may 
be shared with the participants who made it possible. In the case of a Bedoop-encoded magazine 
20 ad, some of the participants may include (1) the photographer or graphic designer who produced 
artwork used in the ad; (2) the advertising agency whose creative talent led to the ad; (3) the 
publisher of the magazine in which the consumer encountered the ad; (4) the service provider(s) 
who provided the transport channel(s) between the consumer and the vendor; and (5) the service 
provider who maintained the server that ultimately linked the Bedoop data to the vendor's web 


10 


15 


25 


site. 


Building Access 

Another use of the Bedoop technology detailed in the 4 104 application (and other 
applications and patents of the present assignee) is to control building access (or facility access, 


(il 
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or room access, etc.) access through a combination of an ID card, Bedoop technology, and 
proximity detection technology. 

The ED card can be a badge or the like having a steganographically-encoded photograph 
of the bearer. The card further includes a proximity ID device, such as an unpowered electronic 
5 circuit that is excited and detected by a radiant field from an associated proximity detector, 
providing a unique signature identifying a particular individual. 

The building can be provided with an image sensor, such as a video camera or the like, an 
associated Bedoop detection system, and the proximity detector. When a user wearing the badge 
approaches, the proximity detector signals the camera to capture image data. The Bedoop 
10 detection system identifies the badge photograph as conveying Bedoop data (e.g., by clues as are 
described in the prior applications, or without such aids), and decodes same. The access control 
system then checks whether the badge ID discerned from the proximity sensor properly 
corresponds to the Bedoop data extracted from the photograph on the badge. If so, access is 

01 granted; if not, the data is logged and an alarm is sounded. 

Lil 

. | 1 5 By such arrangement, premises security is increased. No longer can proximity-based 

1 ™ access badges be altered to substitute the picture of a different individual. If the photo is 

□ swapped, the proximity system ID and the embedded photo data will not match, flagging an 
\ s i unauthorized attempted access. 

The same principles are applicable in many other contexts - not limited to RF-based 
ii 20 proximity detection systems. For example, the data decoded from the photograph can be 

compared against other forms of machine-sensed personal identification. These include, but are 
not limited to, bar code IDs, mag-stripe ED cards, smart cards, etc. 

As known from the prior applications, there are a great number of techniques by which 
the encoding of Bedoop digital data carybe performed. The present assignee's prior application 
09/127,502 (filed July 31, 1998), for example, shows techniques by which very fine lines can be 
printed on a medium to slightly change the medium's apparent tint, while also conveying digital 
data. Commonly-owned application 09/074,034, filed May 6, 1998, details how the contours of 
printed imagery can be adjusted ta convey digital data. (That technique can be applied to printed 
text characters, as well as the line art imagery particularly considered.) The assignee's patent 
30 5,850,481 details how the surfac/e of paper or other media can be textured to convey optically- 
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detectable binary data. The assignee's patents 5,841,886, 5,809,160 and 5,862,260 detail various 


\ techniques for steganoep^phically encoding photographs and other imagery. 

Some watermarking techniques are based on changes made in the spatial domain; others 
are based on changes made in transformed domains (e.g., DCT, wavelet). One example of the 
5 latter is shown in patent 5,930,369. Watermarking of printed text can be achieved by slight 
variations to character shape, character kerning, line spacing, etc. 

Data glyph technology, as detailed in various patents to Xerox, is usable in many of the 
applications detailed herein. Bar codes, too, may also be used. 

The foregoing is just a gross under-sampling of the large number of watermarking 
10 techniques. The artisan is presumed to be familiar with such art, all of which is generally suitable 
for use in the applications detailed herein, 
j j To provide a comprehensive disclosure without unduly lengthening this specification, 

applicant incorporates by reference the patents and applications cited above. 
□J Having described an illustrated the principles of my inventions with reference to specific 

jjj 15 embodiments, it will be recognized that the principles thereof can be implemented in many other, 



different, forms. Moreover, the particular combinations of elements and features in the above- 
detailed embodiments are exemplary only; the interchanging and substitution of these teachings 
with teachings in the incorporated-by-reference applications are also contemplated. 


